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METHOD AND APPARATUS TO INTERACT WITH A MACHINE UTILIZING 

GESTURES 

The present application incorporates by reference the disclosures in the following 
co-pending applications: 

1. U.S. patent application serial no. 09/913,398, entitled "METHOD AND 
DEVICE FOR MONITORING AN ELECTRONIC OR COMPUTER 
SYSTEM BY MEANS OF A FLUID FLOW", filed August 10, 2001. 

2. U.S. provisional application serial no. 60/368,602, entitled "A METHOD 
AND APPARATUS FOR POINTING AND CLICKING FUNCTIONS FOR 
CURSOR TRACKING IN GUI ENVIRONMENT", filed March 29, 2002. 

3. U.S. provisional application serial no. 60/378,561, entitled "METHOD AND 
DEVICE FOR PROVIDING INPUT INTERFACE FOR THE AVIONICS 
AND AEROSPACE APPLICATON", filed May 6, 2002. 

4. U.S. provisional application serial no. 60/378,573, entitled "METHOD AND 
APPARATUS FOR MONITORING AN ELECTRONIC OR COMPUTER 
SYSTEM BY MEANS OF A FLUID FLOW UTILIZING OPTO- 
ELECTRONIC CONVERSION DEVICES", filed May 6, 2002. 

5. U.S. provisional application serial no. 60/402,994, entitled "METHOD AND 
APPARATUS FOR IMPLEMENTING USER INTERFACE CONTROL, 
AND NATURAL LIGHT SOURCE USAGE FOR SUCH CONTROL", filed 
August 12, 2002. 
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6. U.S. provisional application serial no 60/417,539, entitled "METHOD AND 
APPARATUS TO INTERACT WITH A MACHINE UTILIZING A FLUID," 
filed October 9, 2002. 

DETAILED DESCRIPTION 

A method and apparatus to interact with a machine utilizing gestures are 
described. In the following description, for purposes of explanation, numerous specific 
details are set forth in order to provide a thorough understanding of the present invention. 
It will be evident, however, to one skilled in the art that the present invention may be 
practiced without these specific details. 

The present application describes an exemplary embodiment for input interfaces. 

This exemplary embodiment may be deployed in advanced mobile telephony, 
mobile computing, assistive inputting environments. It proposes that a user is enabled to 
run all the functionalities typically found on a mouse (or other UI navigation system) by 
various user interaction including breathing (e.g., air flow) and gestures utilizing ambient 
light with no need for complex commands or to "change the tool", such as, on a mouse, 
moving a finger from the button to the wheel in order to get a satisfactory scrolling, for 
example through web pages or long documents. Continuity and ease of interaction may 
thus be improved. 

In an exemplary embodiment, transitions between motion and scrolling, or 
differentiation between pointing and clicking may be smooth, progressive, in the same 
breath, or in a continuous gesture. 



According to an exemplary embodiment, an interface device is optimized by 
utilizing ambient light. 
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AMBIENT LIGHT 

A substantial energy saving is achieved by reducing the powering of artificial 
sources of light such as LEDs or lasers by using ambient daylight to complement or 
replace artificial sources. 

The present embodiment applies to any kind of ambient light, including any sort 
of fluorescent and/or incandescent lights, thanks to suited light detectors and specific 
windows opened in the casing of the device, close to detectors (3 .a & b.). Thus, the 
present embodiment allows to avoid or at least reduce the piping of ambient light to 
detectors. 

The present embodiment allows the four light detectors (clockwise: left, down, 
up, right motion) to be illuminated at their rest position, the stress breath coming to 
decrease the amount of light to be recei ved by the detector, possibly up to darkness. 

An artificial source of light is embedded in the device (preferably LED), so to 
complement the ambient light for usage by night or in insufficient light, thanks to an 
averaging of the amount of light detected (cycling of detectors), to be processed by 
microcontroller. This artificial light can be piped (fiber optics...) or reflected to the 
detectors (3. a.). 
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Detectors are angled (3. c), in order to get the best ambient illumination in either 
ways of interfacing (air or gesture, see below), regardless of the position of the device, 
supposedly close to the user's chin or put on a surface. 

INTERFACING USING GESTURES 

The use of ambient light makes it possible, with a single device, to interface either 
by means of air flow (i.e., the user breathing) or gesture. 

Indeed, the user can run all the functionalities of a mouse by breathing. In line 
with this, and notably given the cuirent enhanced recourse to ambient light, the user can 
input computers (GUI) and electronic appliances by means of gesture as well. For 
example, the device, generally part of a headset, head-mounted device, or any other 
wearable apparatus enabling the user to breathe into it, can be put down on a desk or any 
surface (possibly after being removed from its usual support, see 3. g. & 4., example of a 
headset), worn on a belt, or inserted in existing workstations, so to be used as a gesture- 
based input interface (5.). This should be noted that the device is preferably wireless 
(eg., Bluetooth). 

That way, movements of the hand will interfere (shade, see 3. f.) with the ambient 
illumination of the device (with no contact with the device itself), and consequently with 
the illumination of the embedded detectors, so to generate an analog sensing, to be 
processed as pointing functionalities (eg., the shade of the hand - or even a single finger 
- positioned directly above a detector, disrupts, possibly up to full darkness as the hand 
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gets closer to the device, the illumination of the "left" detector, thus generating a motion 
of a cursor towards the left, on a monitor... all combinations being possible so to get 
diagonals, while acceleration is provided through processing scalability of the electric 
signals that result from the sensing), whereas Boolean functions such as clicking will 
preferably result from fast movements (brief breaks in detectors' illumination, eg., all 
detectors' illumination disrupted within a short period of time is processed as a single 
click, left button of a mouse if the disruption starts on the left - left-hand detector all 
detectors' illumination disrupted within a short period of time, first from left to right, then 
the opposite is processed as a double click, left button, while all detectors' illumination 
disrupted within a short period of time is processed as a single click, right button of a 
mouse if the disruption starts on the right... A "double-click", "right button" could thus 
be processed as a held click, etc.). The latter are just processing options, insofar as the 
sensing on its own allows to work out a variety of functionalities on the processing side, 
according to usage patterns. 

In an alternate embodiment, buttons on the device can handle the clicking 
functionalities by pressing them. However, such design would provide a minimal-contact 
type of interaction, allowing a high level of protection in extreme environments 
(industrial workstations, for example, in which the device could be put down under an 
airtight, waterproof "bell jar"). Nevertheless, a button can be added on the casing of the 
device to switch between air and gesture-based interaction. 
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Different design options come to enhance the user experience, such as, on the 
ergonomics side, position the "up-sensor" window towards the top of the device (3. e. & 
d.), allowing an easier and more effective gesture recognition, or the way detectors are 
angled to optimize their illumination by the ambient light. 
SEGMENTS DESIGN 

The present embodiment uses four segments in place of eight These non- 
transparent (non conductive to light, full bandwidth) segments can be stressed in two . 
directions, either by the user blowing out (expiring) or sucking in (inspiring), enabling a 
reduction of their number. The segments are, assembled together in the form of a flexible 
comb, possibly holding springs or other mechanical systems able to stabilize the 
segments' rest position (2. d.). Their free end travel is quite restricted (2. a.). 

Channels are specifically designed to channel the air flow from inlet/ outlets 
opened on the top of the device up to those of the bottom by optimizing the stress breath 
on segments, however allowing a logical visual positioning of the top outlets (sections 1. 
a to 1. d.) and a manufacturing/ assembly process. The segments are positioned in 
relatively (no friction between segments and walls) airtight compartments (2. c.) that 
make the vacuum/ "sucking in" stress easier by minimizing the volume of air, and that 
make the stress breath/ "blowing out" action more variable, notably insofar as 100% of 
the air expired comes to stress the segment until it reaches a position that allows the air to 
escape by the inlet/ outlet opened on the bottom of the compartment and device. 
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The segments allow, at their rest position, the detectors to be illuminated by 
ambient light and possibly, when needed, by complementary (preferably piped) artificial 
sources (2.b.). The stress breath comes to make the segments move so that they will: 

- user blowing out: progressively reduce the amount of light able to hit the 
detectors. This results in an analog sensing is able to generate pointing 
functionalities such as movement (cursor, pointer... ), scrolling, zooming, 
according to processing. 

- user sucking in: allow switch-type contacts, thanks to (at least part of) their 
surface being conductive and brushes, wires, or pins. A variety of designs can be 
implemented so to allow different sort of switches. This results in a digital 
sensing able to generate clicking functionalities and similar interactions 
(selection...). This combination of expiration and inspiration, within a short, fast 
sequence, can be used to generate "Boolean" functionalities such as (double) 
clicking. 

The above described processing pattern is an exemplary embodiment, which can 
translate into various programs, including different modes of cycling, checking (IRQ...), 
pulsing the light sources and detectors, several thresholds, ranges, signal levels and 
incrementation, conditions, in order to comply with existing computing environments 
(e.g., ports, interfaces, drivers, GUI functionalities, applications) or new ones as well, 
such as "3D" (three dimensional) applications. 
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Lastly, it should be noted that the exemplary processing pattern can apply to a 
variety of sensors, in addition to those previously described, in order to enable a Human- 
Computer (or Human-to-electronic-appliance) interaction, more precisely an inputting. 

The figures below also show a diagrammatic representation of machine in the 
exemplary form of a computer system within which a set of instructions, for causing a 
machine to perform any one or more of the methodologies discussed above may be 
executed. In alternative embodiments, the machine operates as a standalone device or 
may be connected (e.g., networked) to other machines. In a networked deployment, the 
machine may operate in the capacity of a server or a client machine in server-client 
network environment, or as a peer machine in a peer-to-peer (or distributed) network 
environment. The machine may be a personal computer (PC), a tablet PC, a set-top box 
(STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a 
network router, switch or bridge, a user interface device capable of interacting with any 
of the above described devices, or any machine capable of executing a set of instructions 
(sequential or otherwise) that specify actions to be taken by that machine. Further, while 
only a single machine is illustrated, the term "machine" shall also be taken to include 
any collection of machines that individually or jointly execute a set (or multiple sets) of 
instructions to perform any one or more of the methodologies discussed herein. 



The exemplary computer system includes a processor (e.g., a central processing 
unit (CPU) a graphics processing unit (GPU) or both), a main memory and a static 
memory, which communicate with each other via a bus. The computer system may 
further include a video display unit (e.g., a liquid crystal display (LCD) or a cathode ray 
tube (CRT)). The computer system also includes an alphanumeric input device 1912 
(e.g., a keyboard), a user interface navigation device (e.g., the user interface device 
discussed above), a disk drive unit, a signal generation device (e.g., a speaker) and a 
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network interface device. 

The disk drive unit includes a machine-readable medium on which is stored one 
or more sets of instructions (e.g., software) embodying any one or more of the 
methodologies or functions described herein. The software may also reside, completely 
or at least partially, within the main memory and/or within the processor during 
execution thereof by the computer system, the main memory and the processor also 
constituting machine-readable media. 

The software may further be transmitted or received over a network via the 
network interface device. 

While the machine-readable medium is shown in an exemplary embodiment to be 
a single medium, the term "machine-readable medium" should be taken to include a 
single medium or multiple media (e.g., a centralized or distributed database, and/or 
associated caches and servers) that store the one or more sets of instructions. The term 
"machine-readable medium" shall also be taken to include any medium that is capable of 
storing, encoding or carrying a set of instructions for execution by the machine and that 
cause the machine to perform any one or more of the methodologies of the present 
invention. The term "machine-readable medium" shall accordingly be taken to included, 
but not be limited to, solid-state memories, optical and magnetic media, and carrier wave 
signals. 

Thus, a method and apparatus to interact with a machine utilizing gestures have 
been described. Although the present invention has been described with reference to 
specific exemplary embodiments, it will be evident that various modifications and 
changes may be made to these embodiments without departing from the broader spirit 
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and scope of the invention. Accordingly, the specification and drawings are to be 
regarded in an illustrative rather than a restrictive sense. 
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